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CHALLENGES  IN  DEVELOPING 
A  PERFORMANCE  MEASUREMENT  SYSTEM 
FOR  THE  GLOBAL  VIRTUAL  ENVIRONMENT 


Introduction 

The  dramatic  transformation  of  America’s  strategic  environment  continues  its  significant  impact 
on  today’s  Warfighter  and  its  demand  for  transfonnations  in  how  we  prepare  for  combat 
operations.  Emphasis  remains  on  shifting  from  deliberate  to  adaptive  war  planning,  and  from 
pennanent  organizations  and  large  hierarchies  to  smaller,  highly  distributed  joint  and  combined 
forces  [OUSD,  2004].  A  distributed  training  environment  that  constitutes  global,  multinational 
networks  of  constructive  computer  simulations,  man-in-the-loop  virtual  simulators,  and  live 
forces  at  instrumented  ranges  is  key  to  achieving  military  performance  objectives  and  meeting 
current  and  future  Warfighter  demands  [OUSD,  2002], 

2.1  Distributed  joint  and  coalition  simulation  development 

As  the  necessity  of  military  commitments  steadily  increases  in  today’s  world,  joint  and  coalition 
training  continues  to  play  a  critical  role  in  Warfighter  preparation.  Currently,  Warfighter  forces 
regularly  participate  in  large-scale  “live-fly”  joint  exercises  such  as  Red  Flag  and  Maple  Flag  to 
meet  training  objectives.  This  type  of  high-visibility  training  event,  often  involving  coordination 
of  multinational  forces,  typically  occurs  several  times  a  year  and  helps  keep  the  Warfighter 
combat-ready. 

The  use  of  equipment,  maintenance,  and  support  personnel  required  to  perform  these  “live-fly” 
events  has  stimulated  a  significant  need  for  simulation-based  training.  Simulation-based  training 
in  a  Distributed  Mission  Operations  (DMO)  environment  is  currently  used  to  prepare  the 
Warfighter  and  augment  live-fly  events  by  providing  the  ability  to  practice  or  leam  skills  and 
techniques  that  can  be  validated  and  sharpened  through  live  training  and  real-world  use.  DMO- 
based  training  fills  the  need  to  “train  the  way  we  intend  to  fight,”  and  has  become  a  critical 
requirement  in  Warfighter  preparation,  enabling  the  Warfighter  to  maintain  combat  readiness 
through  joint  and  coalition  mission  rehearsal  in  operationally  realistic  environments. 

Simulation  technology  today  allows  Warfighters  to  participate  in  a  continuous  training  cycle  and 
maintain  a  high  state  of  combat  readiness  by  using  cost-effective  simulation  alternatives  in 
conjunction  with  live-fly  operations  and  training  missions.  The  rapid  advancement  of  networking 
technologies,  the  increase  in  bandwidth  capabilities,  and  the  continued  improvement  of  protocol 
standards/architectures  such  as  Distributed  Interactive  Simulation  (DIS),  High  Level 
Architecture  (HLA),  and  the  Test  and  Training  ENabling  Architecture  (TENA)  have  all 
contributed  to  an  environment  where  large-scale,  multi-force  DMO  joint  and  coalition  exercises 
have  become  a  reality. 

Current  development  and  integration  of  live,  virtual,  and  constructive  systems  for  training, 
mission  rehearsal,  and  operations  support  has  emphasized  the  need  for  more  complete  simulation 
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network  interoperability  among  joint  and  coalition  forces.  Specifically,  improvements  in  protocol 
standardization  and  acceptance  of  comprehensive  performance  measurement  systems  stimulate 
interoperability  and  help  to  validate  the  return  on  investment  (ROI)  that  joint  operations  provide. 
With  the  change  in  training  scope  from  traditional  to  transitional  training,  including 
environments  such  as  special  operation  forces,  urban  operations,  and  joint/coalition,  measuring 
human  performance  gains  in  these  environments  is  critical  if  we  are  to  understand  our 
Warfighters’  readiness  levels. 

The  Need  for  a  Robust  Performance  Measurement  System 

The  Air  Force  Research  Laboratory  in  Mesa,  AZ  (AFRL/Mesa),  a  scientific  organization 
chartered  to  implement  and/or  evaluate  methods  to  improve  Warfighter  readiness,  has  been 
commissioned  to  do  human  performance  assessment  research  with  respect  to  United  States  Air 
Force  Mission  Essential  Competencies  (MECs).  One  of  the  primary  goals  of  this  research  was  to 
investigate  the  ability  to  assess  Warfighter  perfonnance  in  a  DMO  environment.  This  capability, 
if  done  properly,  would  allow  the  research  division  to  collect  data  on  any  number  of  diverse 
studies. 

That  initial  line  of  research  resulted  in  a  DIS  compatible  tool  capable  of  collecting  objective  data 
on  the  DMO  assets  within  the  local  area  network  (LAN)  site  at  that  division.  As  an  evaluation 
tool,  the  proof-of-concept  Performance  Effectiveness  Tracking  System  (PETS)  emphasized 
tracking  human  performance  data  in  an  empirical  (i.e.,  scientific)  manner  for  formally  evaluating 
training  techniques  and  technologies  (Schreiber,  Watz,  Bennett,  &  Portrey,  2003;  Watz,  Keck,  & 
Schreiber,  2004;  Watz,  Schreiber,  Keck,  McCall,  &  Bennett,  2003). 

The  early  successes  of  this  system  has  allowed  AFRL/Mesa  to  automatically  capture  and  store 
kill  ratios,  weapons  employment,  and  other  skill-related  metrics  on  over  400  fighter  pilots 
participating  in  multi-player  DMO  exercises  at  AFRL/Mesa’s  site  since  January  2002.  The 
usefulness  of  this  empirical  data  facilitated  a  number  of  recent  human  performance  DMO  studies 
(Gehr,  Schreiber,  &  Bennett,  2004;  Portrey,  Schreiber,  Stock,  &  Schvaneveldt,  in  press; 
Schreiber,  Bennett,  Schurig,  &  Gehr,  in  press;  Schreiber,  Watz,  &  Bennett,  2003;  Stock, 
Schreiber,  Symons,  Portrey,  &  Bennett,  in  press)  and  data  collaboration  and  study  efforts  from 
both  industry  and  academia.  Additional  AFRL/Mesa  studies  are  either  planned  or  currently  in  the 
data  collection  stage  (e.g.,  DMO  studies  on  skill  decay  and  transfer).  Due  to  the: 

(a)  usefulness  of  this  data  for  studies, 

(b)  an  unanticipated  piqued  Warfighter  interest  in  its  potential  for  feedback,  and 

(c)  expanding  DMO  exercises  outside  of  local  area  network  (LAN)  environments  (e.g., 

joint  and  coalition), 

this  measurement  capability  was  then  sought  by  AFRL/Mesa  for  DMO  exercises  involving  assets 
outside  the  LAN — essentially  an  expanded  scope  of  the  original  human  performance  assessment 
research. 


2 


Additionally,  this  proof-of-concept  assessment  tool,  PETS,  was  identified  by  the  United  States 
Air  Force  (USAF)  Air  Combat  Command  (ACC)  as  a  Category  1  Advanced  Technology 
Demonstration  project  potentially  capable  of  being  instituted  across  the  USAF  for  evaluating 
individual  and  team  skill.  If  successful,  this  system  would  be  standardized  and  become  a 
primary  piece  of  DMO  technology  to  assess  the  Air  Force’s  MECs  (Colegrove  &  Alliger,  2002). 
These  new  requirements  reduce,  but  do  not  eliminate,  the  emphasis  on  empirical  data.  Rather, 
these  broadened  requirements  increase  the  emphasis  to  collect  data  on  any  entity  involved  in  a 
DMO  network  and  report  performance  metrics  as  feedback  (i.e.,  an  increased  emphasis  on  a  new 
“observational”  measurement  capability)  to  Warfighters  and  their  instructors.  To  fulfill  these 
new,  larger  objectives,  a  different  architecture  was  required. 

2.1  Performance  Effectiveness  Tracking  System  vl.6 

The  PETS  software,  developed  at  AFRL/Mesa  in  2000-2001,  was  designed  as  a  proof-of-concept 
human  performance  measurement  system  that  could  collect  over  80-100  “core”  air-to-air  and  air- 
to-ground  combat  perfonnance  measures  in  real-time  from  a  distributed  network  using  the  DIS 
protocol.  This  original  PETS  system  architecture  is  actively  used  to  provide  a  number  of 
previously  unavailable  objective  measures  which  significantly  increase  the  effectiveness  and 
quality  of  DMO  training  and  research.  However,  due  to  its  origin  as  a  proof-of-concept 
application,  some  limitations  to  its  extensibility  existed. 

The  original  proof-of-concept  application  was  designed  to  gather  empirical  data  within  the  LAN 
DMO  at  AFRL/Mesa,  and  therefore  it  could  not  be  expanded  to  meet  the  growing  need  for 
virtual  joint  and  coalition  exercises.  As  a  result  of  inherent  variability  in  different  instructor 
operator  stations  (IOS)  and  network  environments,  PETS  lacked  the  flexibility  and 
configurability  needed  for  use  at  other  sites,  making  any  potential  distribution  of  the  PETS 
software  difficult,  (i.e.,  would  require  customized  patches  or  site  modifications).  Inevitably, 
limitations  in  the  original  design  led  to  the  first  PETS  system  not  being  able  to  reach  its  full 
potential  as  a  highly  distributed  application  capable  of  capturing  common  perfonnance  data 
across  DMO  installations  (Colegrove  &  Alliger,  2002).  To  collect  data  on  any  entity  on  multiple 
DMO  networks,  the  architecture  and  programs  must  be  robust  enough  to  be  “shrink-wrapped”  or 
have  simple  click  to  install/run  capability. 

2.2  Performance  Effectiveness  Tracking  System  Squared  (PETS2) 

PETS2,  beta  developed  in  2003-2004,  was  designed  to  address  the  flexibility  and  configurability 
issues  inherent  in  the  original  proof-of-concept  version  (Berry,  2004).  More  specifically,  we 
wanted  an  architecture  that  went  beyond  the  previously  “empirical-only”  LAN  architecture  to  a 
more  robust  and  flexible  wide-area  “PETS  empirical”  and  “PETS  observational”  architecture, 
hence  the  “squared”  nomenclature.  PETS2  is  also  driven  by  the  requirements  for  increased 
Warfighter  performance  feedback  and  higher  level,  aggregate  measurement  capabilities.  A  high- 
level  view  of  the  PETS2  design  is  illustrated  in  Figure  1 . 
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Figure  1.  PETS2  Architecture 


The  PETS2  project  is  designed  as  a  modular,  multi-threaded  application,  capable  of  robustly 
handling  high  volumes  of  network  entities  (Watz,  Schreiber,  Keck,  McCall,  &  Bennett,  2003). 
The  preliminary  version  of  this  system  is  capable  of  handling  several  hundred  entities  within  a 
DMO  environment.  It  provides  a  number  of  interfaces  used  to  customize  real-time  informational 
graphic  displays,  such  as  a  force  demographic  summary.  This  system  also  employs  a  multi¬ 
tiered  lookup  system  for  additional  user  supplied  internal  state  data  that  may  be  unavailable  on 
the  network,  thus  making  it  more  interoperable  to  any  network  by  removing  the  dependency  on 
custom  data  requests. 

PETS2  development  has  been  extended  to  the  team,  package,  and  force  levels.  Therefore,  the 
system  has  the  ability  to  calculate  measurements  (see  Table  1)  at  the  team,  inter-team  (package), 
and  teams-of-teams  (force)  levels,  which  can  theoretically  extend  the  potential  measurement 
capabilities  of  PETS2  up  to,  and  including,  objective  data  at  the  command  and  control  (C2)  level. 

Since  this  new  architecture  follows  all  entities,  PETS2  is  able  to  evaluate  overall  mission 
performance  on  teams  including  both  man-in-the-loop  and  of  computer-generated  forces  (CGF), 
allowing  the  trainer  to  assess  the  entire  picture  from  both  the  friendly  and  enemy  perspective. 
PETS2  provides  a  default  implementation  of  team,  package,  and  force  objects,  and  the  capability 
exists  to  build  upon  the  base  implementation  as  measurements  are  developed  to  assess  higher 
level  competencies. 
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Table  1  Types  of  DMO  objective  training  effectiveness  data  currently  available  in  the  initial 
PETS2  implementation. 


OUTCOME 

MEASURES 

PROCESS 

MEASURES 

MUNITION 

EMPLOYMENT 

Bombers  reaching 

2  mn  of  target 

Time  each  entity  spent 
within  MOR 

Who  shot  what  type  of  weapon  at  whom 

Minimum  distance 
to  target  bombers 
achieved 

Time  each  entity  spent 
within  MAR 

Result  (hit,  miss,  and  some  types  of 
misses).  Also  records  distance  of  misses 
for  missiles.  For  bombs,  records  left/right 
&  long/short  error  distances 

Fratricides 

Time  each  entity  spent 
within  N-pole 

2D  and  3D  range  of  shot 

Mortalities 

Chaff/Flare  usage 

Altitude  at  pickle 

Enemy  fighters 
killed 

Missile  Clear  Avenue  of 
Fire,  measured  by  angles 

F-pole  for  hits  and  misses 

Enemy  strikers 
killed 

G-load  at  pickle 

A-pole 

Missiles  fired  that 
resulted  in  a  kill 

Wingman  position  in 
relation  to  lead 

Loft  angle  at  pickle 

Scenario 

demographics  (e.g., 
number  of  threats 
presented,  etc.) 

How  often  and  for  how 
long  flight  “steps-on” 
one  another  when  talking 
on  mike 

Mach  at  pickle 

PETS2  was  designed  from  the  ground  up  to  support  DIS  standard  data  (currently  DIS  2.04)  and 
RPR1.0.  Although  this  may  limit  the  amount  of  special  (i.e.,  non-standard)  data  that  a  particular 
simulator  emits,  it  allows  PETS2  to  work  with  any  simulator  that  conforms  to  the  standards. 
However,  to  support  more  complex  measurements,  PETS2  has  an  array  of  user  input  screens  that 
allow  configuration  of  non-standard  data  such  as  weapons  load  and  various  initial  entity  state 
conditions.  PETS2  is  designed  as  a  “horizontally  integrated”  application  that  provides  as  much 
vertical  depth  as  the  protocol  and  user  provided  input  allow. 

To  help  PETS2  take  advantage  of  non-standard  network  data,  PETS2  has  the  capability  to  add 
custom  entity  modules  that  can  handle  custom  information  packets  “passed  through”  the  network 
via  the  DIS  or  RPR  protocol  (such  as  DIS  Set  Data  PDUs,  etc.).  These  custom  entity  modules 
are  physically  separate  from  the  PETS2  core  and  can  be  added  or  removed  as  the  functional  need 
or  security  requirements  pennit.  For  example,  if  a  particular  site  has  a  new,  cutting  edge  flight 
simulator  operating  at  a  higher  classification  level,  that  site  can  create  a  custom  entity  module 
that  is  kept  and  implemented  only  at  their  site  for  collection  of  special  measurements  while 
standard  measurements  are  collected  at  all  other  locations  on  the  network. 

Based  on  a  user-defined  rule  set,  PETS2  can  be  configured  to  use  custom  entity  modules  for 
specific  entities  on  the  network.  For  example,  this  could  allow  participants  in  a  joint  or  coalition 
exercise  to  capture  additional  data  on  certain  entities  on  a  site-by-site,  entity  type,  or  force 
affiliation  basis. 
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Issues  Observed  during  Virtual  Exercises 


Joint  and  coalition  exercise  environments  usually  consist  of  diverse  simulation  systems  and  a 
wide  array  of  internal  and  external  networks.  This  diversity  limits  the  effectiveness  of  a 
performance  measurement  system  in  providing  information  to  the  units  and  command  staff. 
Performance  outcomes  measured  are  inversely  proportionate  to  the  number  of  simulation 
systems  and  the  complexity  of  the  teams  within  the  exercise  environment  (see  Figure  2). 


Complexity  Opportunities 


Figure  2  Performance  Measurement  Complexity 

As  previously  mentioned,  each  participant  is  operating  different  simulation  systems  which,  in 
real-world  situations,  often  have  different  DIS  data  capabilities,  DIS  enumerations,  and  custom 
(non-standard)  PDU  information.  Some  of  the  main  issues  that  were  observed  during  joint  and 
coalition  exercises  are  described  below.  These  issues  play  a  big  part  in  whether  or  not  a 
performance  measurement  system  can  accurately  report  outcome  measures  from  such  exercises. 

•  Participant  Capabilities 

•  Interface  and  Enumeration  Standards 

•  Exercise  Protocols,  or  Rules  Of  Engagement  (ROE) 

2.1  Participant  Capabilities 

Although  the  protocols  are  standard,  due  to  real-world  requirements,  each  simulation  system 
might  be  different  in  the  variations  of  data  packets  that  are  emitted  or  handled.  For  example, 
interoperability  is  frequently  adversely  affected  by  missing  or  incomplete  DIS  PDUs,  incorrect 
DIS  enumerations,  or  non-standard  data  such  as  proprietary  voice  data  implementations,  or  use 
of  unsupported  tactical  data  links.  Better  use  of  network  protocols  and  improved  interoperability 
awareness  is  needed  to  present  a  clear  picture  for  performance  measurement  of  all  participants 
within  the  exercise.  Most  simulations  systems  output  some  degree  of  non-standard  or 
incomplete  data  during  an  exercise;  while  standards  exist,  installations  do  not  strictly  adhere  to 
them.  Non-adherence  simply  should  be  resolved  across  all  DMO  installations,  not  just  for 
performance  measurement,  but  also  for  improved  large-scale  exercise  interoperability. 
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In  addition  to  the  “non-standard”  problem,  simulators  also  may  have  inconsistency  in  models  and 
platforms.  An  example  of  inconsistent  model  that  was  observed  in  various  coalition  operations 
included  inconsistency  of  flight  models  such  as  AIM-9  missile  perfonnance;  some  had 
unrealistic  ranges  or  maneuvers.  This  inconsistency  can  significantly  affect  simple  perfonnance 
measurements  such  as  shot/kill  ratios. 

2.2  Interface  and  Enumeration  Standards 

There  needs  to  be  clear  standards  regarding  DIS  header  information  and  enumerations.  Accurate 
identification  of  platform  models  is  essential  to  utilizable  perfonnance  measurement.  Entity 
marking  string  usage  was  inconsistent  among  the  simulation  systems  adding  complexity  to  entity 
identification  within  the  performance  measurement  system.  The  complexity  significantly 
increases  the  time  to  analyze  the  data  for  entity  performance  and  outcome  measures.  The 
Distributed  Mission  Operations  Center  (DMOC)  has  instituted  an  Entity  Marking  requirement 
for  virtual  flag  exercises  where  if  an  exercise  participant  is  flying  a  friendly  aircraft  whose 
mission  appears  in  the  air  tasking  order  (ATO),  then  the  simulation  must  place  an  abbreviation 
on  the  aircraft  call  sign  in  the  Entity  Marking  field  of  the  Entity  State  PDU  (Berry,  2004).  This 
is  one  example  that  can  ensure  that  the  correct  measures  correspond  to  the  correct  entity. 

Inconsistent  timeout  values  are  another  major  issue  that  can  affect  outcome  measures.  For 
example,  many  sites  will  have  different  timeout  values  for  air  and  ground  entities.  This  common 
change  done  to  improve  the  perfonnance  of  the  network  can  hinder  performance  measurements 
by  impacting  the  fidelity  and  latency  of  the  output.  Due  to  the  timeout  differences,  single  entities 
could  show  up  as  multiple  entities  within  the  same  exercise  confusing  the  output  results. 

Missing  or  incomplete  network  data  typically  plagues  a  coalition  exercise.  Many  simulation 
developers  will  opt  to  only  include  data  that  is  significant  to  their  own  site  or  purposes,  without 
regard  on  how  this  affects  their  overall  visibility  in  a  large-scale  DMO  exercise.  For  example, 
many  simulators  do  not  emit  shooter/target  information  for  bombs.  This  causes  a  problem  when 
determining  performance  at  time  of  launch,  measuring  data  during  the  fly-out,  or  doing  real-time 
performance  comparisons  with  other  simulators  on  the  network.  These  omissions  can  cause  air- 
to-ground  performance  measurements  to  be  incomplete  or  nonexistent. 

2.3  Exercise  Protocols  or  Rules  of  Engagement 

Protocol  issues  such  as  the  use  of  shields  and  regeneration  rules  need  be  set,  agreed  upon,  and 
followed  consistently  to  ensure  that  common  measurements  such  as  shot/kill  ratios  are  recorded 
accurately.  It  is  extremely  difficult  to  assess  overall  mission  performance  when  the  “fair-fight” 
issue  is  compromised.  For  example,  in  a  recent  coalition  simulation  event,  a  particular  site 
determined  that  they  would  use  shields  on  their  aircraft  in  order  to  maximize  time  in  the 
simulation  event.  Because  they  were  perceived  as  invincible  by  both  themselves  and  other 
players,  this  caused  the  pilots  and  other  participants  to  not  respond  in  a  realistic  manner,  thereby 
compromising  the  training  quality  of  the  data.  Worse  yet,  any  shots  taken  and  detonated  on  an 
entity  with  shields  severely  (and  negatively)  impact  the  most  important  metric — kills.  To 
accurately  train  and  assess  perfonnance  at  the  C2  level,  all  players  in  an  exercise  must  adhere  to 
common  rules  that  map  directly  to  real-world  situations. 
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One  simple  overarching  rule  must  govern  all  simulation  exercise  protocol — realism.  That  is,  if  it 
is  not  possible  in  war,  the  console  operator(s),  mission  directors,  and  simulation  operators  should 
not  be  allowed  to  do  it  in  a  simulation  (e.g.,  use  shields,  reload  weapons  mid-air,  “freeze”  or 
“static”  fuel  bum,  etc.).  If  realism  is  emphasized  when  perfonnance  is  calculated  for  an  exercise, 
the  results  have  much  higher  credibility  and  relationship  to  that  of  an  actual  battle — after  all,  the 
original  impetus  behind  training  in  DMO  was  to  translate  performance  gains  into  actual  battle! 
The  paradox  is  that  the  DMO  community  continually  strives  for  realism  in  all  technology  aspects 
(e.g.,  visuals,  flight  models,  missile  models,  CGFs),  but  the  DMO  community  appears  to  largely 
disregard  exercise  realism  from  the  IOS.  Of  course,  occasional  uses  of  shields  or  other  functions 
may  be  highly  desirable  during  early  stages  of  training,  but  those  non-realistic  scenarios  must  be 
the  exception,  not  the  norm. 


Standards  from  simulation  community 

To  effectively  study  human  and  team  performance  within  the  DMO  environment,  simulation 
systems  need  to  ensure  that  standards  are  adhered  to.  However,  each  individual  simulator 
provides  a  certain  degree  of  non-standard  simulation  data.  This  is  because  simulation 
environments  are  typically  developed  to  address  a  specific  training  problem,  such  as  weapons, 
flight,  or  instrument  training.  As  mentioned  earlier,  nonstandard  data  is  a  problem  that  will  need 
to  be  dealt  with  to  better  meet  joint  and  coalition  interoperability  objectives. 

In  the  process  of  following  current  network  protocols,  the  extension  of  standardized  data  by 
using  existing  and  new  capabilities  of  the  DIS/HLA  standards  would  be  beneficial  in  establishing 
a  more  concise  picture  of  performance  for  individuals,  teams,  and  joint  and  coalition  exercises. 
The  main  purpose  of  this  extension  would  be  to  provide  additional  data  needed  to  perfonn  C2- 
level  Warfighter  perfonnance  assessments  and  training,  which  is  a  significant  reason  distributed 
simulation  environments  were  created  in  the  first  place. 

A  significant  portion  of  the  extended  data  would  be  in  the  form  of  “internal”  state  data,  examples 
of  which  include,  but  are  not  limited  to,  switch  positions,  display  modes,  radar  modes,  radar 
tracking  lists,  weapons  load,  fuel  status,  fuel  bum  rate,  throttle  position,  targeting  information  for 
bombs,  and  others,  all  of  which  are  not  currently  part  of  the  DIS  standard  or  base  HLA  RPR 
Federation  Object  Model  (FOM).  This  type  of  infonnation  is  not  needed  to  stimulate  the  image 
generators  or  simulation  environment  models,  but  rather  to  provide  a  performance  analysis 
system  such  as  PETS2  insight  into  what  the  simulation  operator  (pilot,  weapons  officer,  etc.)  is 
doing.  This  information  helps  to  map  actual  performance  to  MECs  and  achieve  ACC’s  vision  of 
assessing  skill  proficiency  across  DMO  installations. 

When  an  entity  is  placed  upon  the  network,  it  could  send  out  an  initial  update  of  all  its  properties. 
During  the  exercise,  any  entity  exposing  additional  properties  would  then  be  required  to  send  an 
update  when  its  internal  state  data  changes.  For  example,  if  an  entity  is  participating  in  a 
bombing  task,  then  targeting  data  would  be  included  in  the  update. 

Having  internal  state  data  as  part  of  an  established  protocol-level  standard  would  enable 
performance  measurement  systems  such  as  PETS2  to  effectively  gather  C  -level  data  and  analyze 


performance  across  multiple  DMO  sites.  Currently,  standardized  measurement  tools  such  as 
PETS2  are  limited  in  collecting  perfonnance  measures  from  multiple  sites,  as  the  enumerations 
or  data  structures  used  to  pass  such  internal  state  data  is  not  consistent  from  site  to  site  (Watz, 
Schreiber,  Keck,  McCall,  &  Bennett  (2003). 

Conclusion 

Future  military  operations  will  be  effects-based,  rely  on  increased  Warfighter  use  of  integrated 
on-demand  sensor  infonnation,  require  more  responsive  time  critical  targeting,  incorporate  a 
growing  arsenal  of  precision  weapons,  and  utilize  expanded  non-kinetic  options,  including 
information  operations.  To  respond  quickly  to  the  dynamic  challenges  of  today’s  environment, 
training  needs  to  be  flexible,  operationally  effective,  and  integrated  with  real-time,  globally 
distributed  mission  rehearsal  and  command,  control,  communications,  computer,  intelligence, 
surveillance,  and  reconnaissance  (C4ISR)  capabilities.  To  do  this,  available  networks  for  mission 
rehearsal,  simulation,  and  just-in-time  training  must  be  continuously  modernized  and  utilized; 
and  perfonnance  metrics  need  to  be  systematic  and  complete  to  improve  operational 
effectiveness,  both  individually  and  collectively. 

This  global  training  model  emphasizes  the  necessity  for  following  strict  standards  such  as  DIS, 
HLA  (with  a  common  FOM  such  as  RPR)  or  TENA,  and  the  need  to  establish  new  standardized 
data  to  provide  better  perfonnance  feedback  to  the  training  and  operational  communities. 

There  are  many  DMO  installations  throughout  the  Modeling  and  Simulations  (M&S) 
community,  and  currently  most  of  these  sites  have  no  method  of  objectively  assessing  the  degree 
or  amount  of  knowledge  transfer  that  takes  place  when  Warfighters  train  in  these  virtual 
environments.  With  the  training  community  developing  performance  assessment  systems  such  as 
PETS2  to  address  joint  and  coalition  training  issues,  there  is  an  increase  need  to  broadcast 
internal  state  information  on  the  simulation  network  for  acquisition  and  analysis.  Providing  a 
dynamic,  capabilities-based  training  for  the  Warfighter  must  be  a  joint  effort  between  the 
simulation,  operational,  and  training  communities  if  it  is  to  succeed  in  today’s  environment. 
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